Process for Cyclic, Interactive Image Analysis, and also 
Computer System and Computer Program for Performing the Process 

Cross-References to Related Applications 

Not applicable. 

Statement Regarding Federally Sponsored Research or Development 
Not applicable. 

Background of the Invention 
[0001] The present invention relates to a process for cyclic, interactive image 
analysis, and also to a computer system and a computer program for performing 
a corresponding process. 

Technical Field 

[0002] Many image processing methods known today operate according to 
the so-called fractal hierarchy, in which the image processing takes place in 
two steps. In a first, segmentation, step the image is segmented in the local 
space, that is, divided into different regions. The segmentation takes place 
according to hierarchy stages, where each pixel of the image represents a 
segment in the lowest hierarchy stage. For the respective next higher hierarchy 
stage, the two neighboring segments are combined which produce the smallest 
increase of the product of the number of pixels and the variance of the color 
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value or gray-scale value for the combined segment. The selection of the 
segments takes place stochastically. Furthermore, an edge treatment of the 
segments takes place, in which, for each edge pixel of a segment, it is tested 
whether it possibly belongs better to the neighboring segment. If necessary, it 
is so allocated. The theoretically highest hierarchy stage consists only of a 
single segment. The heirarchy stages have between them the property that 
segment boundaries of a heirarchy stage remain maintained in the heirarchy 
stage situated therebelow; that is, the segments are always proper subsets of the 
segments, situated thereabove, of the higher heirarchy stage. 
[0003] In a succeeding classification step, a semantic classification takes 
place, that is, objects with a name are produced from the formed segments. 
Here rules are given for the formation of objects from a set of predefined 
properties of the segments in all heirarchy stages. The rules are formulated as 
fuzzy rules. 

[0004] Corresponding image processing methods are described in, for 
example, German Patent Documents DE 196 25 906 Al, DE 197 05 017 Al 
and in International Patent Document WO 99/40539. The processes concerned 
there run either more or less completely automatically or interactively. 

[0005] The main problem in image analysis is that the images to be analyzed 
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can have completely different, initially unknown and in part highly complex 
image data, and in addition it is not initially known which image data contained 
in the image to be analyzed is important to the user. For universal 
applicability, the known image processing methods therefore require both a 
very large programming cost in their production and also comprehensive 
experience on the part of the user in the choice of suitable segmentation and 
classification rules. 

Summary of the Invention 
[0006] The present invention has as its object to provide an image analysis 
process which can cyclically and interactively approximate the wishes and 
ideas of a user, without the user having to have detailed knowledge of the 
image processing itself. 

[0007] This object is attained by a process for cyclic, interactive image 

analysis with the steps of: 
[0008] a) a selection by the user of image regions of interest in the image, 
[0009] b) performance of preprogrammed transformations on the image 

at all image points or at a selection of image points, 

[00010] c) performance of normalization of the individual transformation 
results, 
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[0001 1] d) division of the whole normalized transformation space into 
classes with the result of a classification in the transformation space with the 
aid of separation rules which are derived from the transformation result values 
in the image regions selected by the user, 

[00012] e) allocation to the individual image points of the classification 
obtained in the transformation space, 

[00013] f) representation of the thus classified image in the image space. 

[00014] The object also is attained by a computer system suitable for the 
performance of the aforementioned process which has devices for the 
interactive input and selection of image regions. The object also is attained by 
computer programs which can be loaded into the memory of a digital computer 
and which has a software code for the performance of the aforementioned 
process with the steps (a)-(f) set forth above when the code runs on the 
computer. 

[00015] The process according to the invention is interactive in the sense that 
the user in step (a) can select or identify image regions in the image which are 
of interest to him or are to be differentiated, and after carrying out the image 
analysis the user is presented in step (f) with a classified image in the image 
space. The user can adjust anew the criteria for the regions, according to his 
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expert opinion. However, no operation by the user is provided for during the 
course of the process according to the invention. 
[00016] The process according to the invention operates cyclically in the 
sense that the steps (a) through (f) or (b) through (f) can be carried out 
successively plural times, until the user is satisfied with the presented result of 
the image analysis. 

[00017] A segmentation admittedly takes place in the process according to the 
invention, as in the state of the art; however, this segmentation is carried out in 
a transformation space. For this purpose, various preprogrammed and stored 
transformations are first performed on the image in all image points, or in all 
image points of a selected region. A normalization of the results of the 
transformations performed in the transformation space takes place thereafter. 
Directly following this, the whole normalized transformation space is divided 
into classes, the separation rules for the classification of the transformation 
result values in the image regions selected by the user being automatically 
derived. The classification obtained in the transformation space is then allocat- 
ed in turn to the individual image points, and thus a classified image corre- 
sponding to the transformation space is thereafter displayed in the original 
image space. 
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[00018] The present invention starts from the precondition that the user 
always has for an image a certain idea of what in the image is to be 
comprehended as belonging to it for his purposes. The user informs the 
computer or the image analysis program of this idea of his own by means of 
regions, which are roughly drawn in and delineated as desired in the image. 
The image analysis program develops its own interpretation of the user's ideas 
based on this interactive setting of the task, and shows the user the developed 
idea. The user can now modify this "machine conception" or "machine idea" 
produced by the image analysis program by further interactive operations, 
through which the image analysis program corrects the "machine conception" 
or "machine idea" according to the user's intention. The user's ideas are 
aligned with the "machine ideas" in an interplay in this manner, so that the 
image analysis program knows at the end what ideas the user has. 
Accordingly, the basic idea of the process according to the invention is a cycle 
within a cyclic process. 
[00019] The formation of the mean and variance of the image brightness or of 
the image color in an image point neighborhood is included under the prepro- 
grammed transformations. A further advantageously preprogrammed transfor- 
mation for the evaluation is a Radon transformation for the effective 
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representation of lines and edges within the image. The preprogrammed 
transformations are used in the process according to the invention as 
completely equivalent, complementary, and orthogonal quantities. 

[00020] The division of the normalized transformation space can preferably 
take place such that those regions in the transformation space are identified in 
which the normalized values of each of the performed transformations 
corresponds to, or comes close to, the normalized values of the same 
transformation in the image regions selected by the user. 

[00021] In the sense of a cyclic process, the user, if he is not yet satisfied with 
the displayed image after a first run of the process, can select further image 
regions, or correct the boundaries of the already selected regions, and then 
allow the whole process to anew run repeatedly. Alternatively, as long as the 
user has not selected any new image regions or corrected the boundaries of the 
selected regions, it is possible for the process to be repeatedly performed on the 
same image with different transformations or different parameters of the 
transformations. 

[00022] The sequences of transformations and normalizations or 

classifications determined according to a single or plural run of the process can 
be stored and subsequently applied to other similar images. 
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[00023] In a further advantageous embodiment example, in addition to the 
displaying of the classified image, additional properties of the classification 
obtained, taking account of the corresponding classification properties in the 
image regions selected by the user, are calculated and indicated to the user. 
Based on these additional properties, the user can decide to what extent the 
"machine conception" or "machine idea" agrees with his own idea. 
Furthermore, the output is possible of numerical data regarding the respective 
associated image regions: for example, the size or number of given regions. 
Corresponding numerical evaluations can then be undertaken automatically for 
similar images, without the user himself having to inspect the similar images. 

Brief Description of the Drawings 

[00024] Particulars of the present invention are explained in further detail 
herein below with reference to the accompanying drawings. 

[00025] Fig. 1 shows a block circuit diagram of a workplace with a 
computer, suitable for carrying out the process according to the invention; 

[00026] Fig. 2 shows a flow chart of the steps taking place in the process 
according to the invention; 

[00027] Figs. 3-10 show graphic illustrations of the intermediate 

results attained in the individual intermediate steps in the process according to 
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the invention, in the example of a cell image; and 
[00028] Figs. 11-13 show graphic illustrations of the results when 

applying the process according to the invention to a simple gray-scale image. 

Detailed Description of the Invention 
[00029] The workplace shown in Fig. 1 substantially consists of a computer 

(2) and an input-output unit (1) attached to the computer. The output unit can 
be a projection screen, a monitor, or a display. The input unit can be a 
conventional keyboard or computer mouse. 

[00030] The computer (2) has, as a hardware component, an image memory 

(3) in which are stored the image to be analyzed and the "machine conception" 
or "idea" of the analyzed image produced by the computer after the 
performance of the process according to the invention. Furthermore, as a 
hardware component, an input memory (4) is provided in which the user's 
inputs are stored. The further modules shown in Fig. 1, "abstraction unit" (5), 
"classification unit" (6), and an "analyzing and control unit" (7) are a 
combination of hardware and software, namely of a working memory, a 
computer processor, and program modules stored in the working memory of the 
computer. 

[00031] The individual steps taking place in the process according to the 
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invention are explained hereinafter with reference to the flow chart in Fig. 2. 
Firstly, an image stored in the image memory (3) is presented to the observer in 
a step (10) by means of the output unit (1). This two-dimensional image in 
general consists of an intensity distribution I as a function of the space 
coordinates x, y. In a step (11) following thereafter, the user selects, by means 
of the input unit (1), image regions of interest to him, or the coordinates of the 
image regions of interest to him. Let coordinates of these image regions or of 
the selected image pixel be (xm, ym). Then individual small image regions, or 
else several large image regions, or the whole image, can be selected. When 
the whole image is selected, the user leaves the further evaluation completely 
to the image analysis program, at least in a first cycle of the process according 
to the invention. 

[00032] In a subsequent step ( 1 2), the computer carries out selected 
transformations Ti on the image in all the image points (x, y). The index i 
identifies the various transformations. Corresponding transformations can be 
the formation of the mean of the gray-scale value (13) of the image in the 
individual image points, or the formation of the variance (14) of the image 
brightness, respectively in an image point neighborhood of the image. The 
concepts gray- scale value and variance of the image brightness are also 
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substitutes here for color brightness and variance of color brightness in the 
different colors. The Radon transformation (15) is a further preprogrammed 
transformation, particularly for the effective representation of lines and edges. 
Besides these three cited transformations, yet other transformations can 
however also be provided which can be selected as desired, or selected by the 
computer itself. 

[00033] In step ( 1 2), a transformation function Ti(x,y) dependent on the local 
coordinates (x, y) is calculated for each of the selected transformations. In a 
normalization step (16) following this, the resulting values of the transforma- 
tions are normalized to the value range [0,1]. Various continuous functions 
can be used for the normalization. The sigmoid function 

y = l/(l + exp(-g* (x ~ x ° } )) 

has been found to be particularly suitable, since it images an unbounded domain of 
values onto the desired value range [0,1]. In a succeeding step (17), a division or 
classification takes place in the transformation space. As long as no further 
information is present, for example due to feedback via a quality function, a rough 
division then takes place of the numerical ranges of the normalized transformation 
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results Ti*(x,y) into three regions, namely small, middle, and large. Moreover, 
separation rules for the separation of the classes are derived for the division or 
classification of the normalized transformation result values Ti*(xm, ym) in the 
image regions selected by the user, taking account of, or with the aid of, these 
normalized transformation results in the image regions selected by the user. 
[00034] In a succeeding step, the classification found in the transformation 
space is referenced or allocated to the individual image points (x, y), and the 
thus classified image is presented to the observer in the image space in a step 
(19), either as an independent image or as a superposed representation of the 
classified image onto the original image. 
[00035] In a succeeding step (20) the user is questioned as to which way the 
process is to be continued, or whether the result of the image analysis already 
agrees with the user's ideas. In a succeeding step (21), the user can effect 
different inputs by means of the input unit (1). If the result of the image 
analysis already agrees with the user's ideas, the parameter settings made 
during the performance of the transformations, normalization, and 
classification are stored in a step (22) and are thus available for an automatic 
application to a later image or to a series of images with similar features. 
[00036] If, however, the user is not yet satisfied with the result of the image 
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analysis, the user can either select additional image regions in the image or 
correct the selection previously made, the evaluation process returning in this 
case to the step "selection of image regions" (11). If however the user inputs 
no new image regions or does not correct the previously input image regions, 
the computer carries out the process anew with changed parameter settings for 
the individual transformations Ti or with altered transformations. In this case 
the program returns to the transformation step (12) described hereinabove. 
After this return either to the selection step (1 1) or to the transformation step 
(12), the process is run through again, until the user is satisfied with the result 
of the analysis. 

[00037] The process described in a rather abstract manner up to now is 
described hereinbelow with reference to Figs. 3-10 for the example of a 
microscope image of a nerve cell. In a very first step, before the image analysis 
proper, the user loads the image of interest into the image memory (3) of the 
computer, so that the image is displayed on the output unit (1). The original 
image of a nerve cell presented on the output unit (1) is shown in Fig. 3. 

[00038] In a succeeding step (Fig. 4), the user selects exemplary image 
regions. As required, the user can allocate names to these selected image 
regions, so that it is clear for the image analysis program whether, for plural 
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selected image regions, these are to be considered as different or as image 
regions which belong together. Image regions with the same name are 
respectively to be considered as belonging together, and image regions with 
different names are to be considered as different. The names of the regions are 
then likewise stored. In the example shown in Fig. 4, the user has made a 
division into three regions: the image background, the cell body, and the 
dendrites. The selection of the corresponding image regions can here take 
place easily with any optional input unit for the image coordinates (mouse, 
roller-ball, etc.), in that the user roughly moves the cursor around the image 
regions to be selected and thus draws boundaries. Where there are regions, 
which are to be classified differently, the user gives each of these regions a 
different name. If on the other hand the regions are not to be classified 
differently, the user gives these regions the same name. 
[00039] Since there are no further user inputs into the system, and the present 
indications of the user are two-dimensional, the evaluation and control unit (7) 
instructs the abstraction unit (5) to apply translationally and rotationally invari- 
ant transformations to this image. Suitable for this purpose are in particular the 
formation of the local mean and local variance of the image brightness or of the 
color brightness for each picture element of the image. Consequently, two 
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transformed intermediate images are formed in the abstraction unit, namely an 
intermediate image of the moving average of the gray-scale value (Fig. 5) and 
an intermediate image of the variance of the gray-scale value over predeter- 
mined regions (Fig. 6). However, these images in the transformation space are 
in the usual manner not presented to the user. 

[00040] It is to be remarked at this point that other suitable transformations, or 
possibly also more than two transformations, can be carried out according to 
the user' indications. For example, if the image regions selected by the user 
were of a linear nature, a Radon transformation would also be carried out as a 
further transformation in the transformation step (2) (Fig. 2). 

[00041] In a succeeding step, automatic normalization now takes place. This 
is appropriate, since the numerical ranges of the transformation results can be 
very different, according to the transformation and the actual image content, 
and the transformation results have to be able to be expressed in a uniform 
relationship to each other. By the normalization, the numerical ranges of all 
the transformations are brought to the numerical range between 0 and 1 by 
means of a sigmoid function. 

[00042] The resulting values of the individual transformations are now 

transmitted from the abstraction unit (5) to the classifier (6), which sets them in 
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a relation to each other. In the above-described case with only two 
transformations, it is a two-dimensional relationship, which can be represented 
imagewise as a two-dimensional histogram. This histogram is shown in Fig. 8, 
in which the mean (m) is plotted toward the right and the variance (v) 
downward. The classifier now determines, with the aid of the user's indications 
which are present, those regions in the transformation space which correspond 
to the user's ideas. For this purpose, there is tested, for example, which 
normalized transformation result values for the various transformations which 
were carried out, in this case formation of the mean and variance, are present in 
the image regions selected by the user. In transformation space, those points 
are then determined to be class centers in which the transformation result 
values agree with the transformation result values in the image regions selected 
by the user. The formation of an idea of the machine now consists in that it 
extends the surroundings of each class center until the class boundaries adjoin 
each other. In this sense, the user's indications are generalized until the whole 
parameter space is filled. The "machine conception" or "machine idea" 
("machine conception" or "machine idea" means the representation which can 
be seen from the viewpoint of the machine or by an observer having the 
machine's knowledge) is shown in Fig. 7 in the transformation space, the 
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respective class centers being marked as black points. This image also is not 
necessary for the observation and evaluation of the result by the user and is 
usually not shown, but is only present in the computer as an intermediate 
image. 

[00043] An alternative to the definition of the class regions consists of 
explicitly representing the mutually adjoining class boundaries by separating 
surfaces (by which n-dimensional surfaces are meant, where the dimension of 
the surfaces results from the number of transformations). Sigmoid functions 
are preferably used for such separating surfaces, such as are also used for the 
normalization of the respective selected transformation space. 

[00044] The classification formed in the transformation space is next again 
referenced or allocated to the individual image points in the image space, and 
the "machine conception" or "machine idea" of the original image thus 
generated is now transferred back to the image memory (3) and presented to the 
user in a suitable manner in the image space, for example as a proper image or 
as a superposition of the original image and the "machine conception" or 
"machine idea". The "machine conception" or "machine idea" in the image 
space is shown in Fig. 9. 

[00045] Ifthe user's idea of the classification of the object does not agree with 
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the presented "machine conception" or "machine idea", the user can input 
further indications for his idea, for example in that further regions are marked 
in the image, or the previous markings are altered. 
[00046] Further inputs by the user preferably serve for the refining or the 
generalization of image regions already input in the sense of hierarchies. Thus, 
for example, the whole region of the dendrites according to a general, 
superordinate representation can be further resolved into the individual 
dendrites. This has the advantage that the search for individual dendrites can 
be restricted to the region of the dendrites, so that (a) computing time can be 
saved, and (b) the accuracy of determination of the individual dendrites can be 
increased, since in this case, for a renewed, further transformation, the whole 
transformation space alone can be exhausted for the determination of the 
dendrites. 

[00047] If the user inputs no further indications, the program assumes that the 
"machine conception" or "machine idea" agrees with the user's idea, and the 
user can input further analysis tasks, for example as regards the size of the cell 
body. The analysis program consequently determines the number of picture 
elements in the class of the cell body and outputs the result to the user after 
suitable scale conversion. 
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[00048] It should be expressly remarked at this point that the procedure 
described hereinabove can be carried out both with an optional number of user 
indications and also with an optional number of transformations. Furthermore, 
optional evaluation functions (determination of the size of regions according to 
length, width, and area, density of the regions, etc.) can be applied to the 
structures processed and classified with the described process. 

[00049] Further information arises in the classifier (6) in addition to the 
"machine conception" or "machine idea" of the analyzed image. A quality of 
this "machine conception" or "machine idea" can also be stated. Considering 
the "machine conception" or "machine idea" in the transformation space (Fig. 
7), it can be recognized that the different class sizes are different and the class 
centers have different distances from one another. The smaller the distances 
between the class centers, the more likely is the risk of a confusion; or the 
greater the distance between the class centers, the smaller is the risk of a 
contusion. These additional data can also be transformed back into the image 
space, in that the associated class distance in the transformation space is 
allocated to each picture element as a parameter. This quality function in the 
image space, which states an opinion concerning the quality of the "machine 
conception" or "machine idea" , is shown in Fig. 10 for the image in Fig. 9. 
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Here dark image points stand for small distances of the class centers in the 
transformation space, and thus for low quality and a high risk of confusion, and 
bright picture elements stand for large distances of the class centers in the 
transformation space, and for high quality and a low risk of confusion. 

[00050] This quality function is transmitted from the classifier (6) to the 
evaluation and control unit (7), and serves there for automatic decision 
concerning the most favorable transformations or respectively, the most 
favorable parameters for these transformations. A further optimizing cycle can 
be automatically initiated here, according to computing time requirements or 
the reaction desires or accuracy desires of the user, in that the program returns 
anew to the transformation step and performs the transformations anew with 
altered parameter settings. 

[00051] A further very simple example, which however well illustrates the 
performance of the process according to the invention, is shown in Figs. 11-13. 
Fig. 1 1 shows the original image, which has a variance of 1.0 above a diagonal 
running from bottom left to top right, and a variance of 1 .4 below this diagonal. 
The mean gray-scale value is constant over the whole image. In the original 
image, only a slight difference in the magnitude of granularity can be 
distinguished with the naked eye. However if, based on corresponding 
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expectations, the observer marks that a difference must be present between the 
two image halves, and gives different names to these two image regions above 
and below the diagonal, the transformations of gray-scale value and variance of 
the gray-scale value are again automatically carried out by the machine, and the 
"machine conception" or "machine idea" shown in Fig. 13 is produced and 
presented to the user. In contrast, the solution of this task would not be 
possible with the known process of fractal hierarchy, since the variance of the 
color value or gray-scale value is only an auxiliary or control quantity in the 
fractal hierarchy, while in the process according to the invention the individual 
transformations to be carried out are considered as equivalent, orthogonal or 
mutually complementary quantities. 
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